Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix flaky tests by disabling automatic plugin acquisition for all tests #2190

Merged
merged 20 commits into from
Jul 19, 2024
Merged

Fix flaky tests by disabling automatic plugin acquisition for all tests #2190

merged 20 commits into from
Jul 19, 2024

Conversation

VenelinMartinov
Copy link
Contributor

@VenelinMartinov VenelinMartinov commented Jul 17, 2024 •
edited
Loading

Disable automatic acquisition globally in the repository - this should ensure we do not use the GH API during tests and make our tests here a lot more robust.

Instead manually install all required plugins in the Makefile as a dependency of the tests. This ensures we only do it once and do not spam the GH API with redundant requests causing test flakes.

I've hit this more than 5 times on separate PRs in the last few days. It's seriously slowing down progress in the repo.

When we do hit the GH rate limit, we have to wait for up to an hour for it to reset.

Skip TestConvertViaPulumiCLI until pulumi/pulumi#16469 is addressed.
Skip TestResourceWithoutID until #2191 is addressed.

Looks like we were using the wrong env variable name...

One of the tests was also absolutely spamming the GH API: https://github.com/pulumi/pulumi-terraform-bridge/actions/runs/9973182117/job/27557985796?pr=2190

Latest failures: https://github.com/pulumi/pulumi-terraform-bridge/actions/runs/9972810153/job/27556840950?pr=2186

fixes #2191

@codecov Codecov
Copy link

codecov bot commented Jul 17, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 6.66667% with 14 lines in your changes missing coverage. Please review.

Project coverage is 60.62%. Comparing base (4ae2ad6) to head (186b824).
Report is 7 commits behind head on master.

Files Patch % Lines
unstable/testutil/installed_providers.go 0.00% 14 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##           master    #2190      +/-   ##
==========================================
- Coverage   60.74%   60.62%   -0.13%     
==========================================
  Files         353      355       +2     
  Lines       46146    46311     +165     
==========================================
+ Hits        28032    28076      +44     
- Misses      16559    16678     +119     
- Partials     1555     1557       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@VenelinMartinov VenelinMartinov changed the title Skip flaky test Skip flaky tests Jul 17, 2024
@VenelinMartinov VenelinMartinov changed the title Skip flaky tests Fix flaky tests Jul 17, 2024
@VenelinMartinov
Copy link
Contributor Author

This might be hitting API rate limits again: pulumi plugin install converter terraform 1.0.18 exit status 255

@VenelinMartinov VenelinMartinov changed the title Fix flaky tests Fix flaky tests by disabling automatic plugin acquisition for all tests Jul 17, 2024
@VenelinMartinov VenelinMartinov marked this pull request as draft July 17, 2024 13:01
@VenelinMartinov VenelinMartinov marked this pull request as ready for review July 17, 2024 13:11
@VenelinMartinov VenelinMartinov requested a review from a team July 17, 2024 13:19
iwahbe
iwahbe approved these changes Jul 17, 2024
View reviewed changes
Copy link
Member

@iwahbe iwahbe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs an export PULUMI_DISABLE_AUTOMATIC_PLUGIN_ACQUISITION in the Makefiles to the Make variable is exposed in the shell. Otherwise LGTM.

@VenelinMartinov VenelinMartinov enabled auto-merge (squash) July 17, 2024 16:57
@VenelinMartinov
Copy link
Contributor Author

Running out of ideas on how to fix TestAccProviderConfig - looks like a problem with directories/PATH?

Where are the pulumi plugins installed and why can't the test find them?

@iwahbe
Copy link
Member

iwahbe commented Jul 18, 2024

Running out of ideas on how to fix TestAccProviderConfig - looks like a problem with directories/PATH?

Where are the pulumi plugins installed and why can't the test find them?

Does the test failure replicate locally?

@iwahbe
Make local test providers available
186b824 
This doesn't appear to be necessary for Mac and Linux, but it does appear necessary for
Windows.
@VenelinMartinov VenelinMartinov merged commit 45e5bcc into master Jul 19, 2024
11 checks passed
@VenelinMartinov VenelinMartinov deleted the vvm/fix_flaky-test branch July 19, 2024 20:18
@mjeffryes mjeffryes added this to the 0.107 milestone Jul 24, 2024
@pulumi-bot
Copy link
Contributor

This PR has been shipped in release v3.88.0.

@pulumi-bot pulumi-bot mentioned this pull request Jul 25, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@iwahbe iwahbe iwahbe approved these changes

@t0yv0 t0yv0 Awaiting requested review from t0yv0

@guineveresaenger guineveresaenger Awaiting requested review from guineveresaenger

Assignees

@VenelinMartinov VenelinMartinov

Labels
None yet
Projects
None yet
Milestone
0.107
Development

Successfully merging this pull request may close these issues.

GH API Rate limitting causing persistent test failures
4 participants
@VenelinMartinov @iwahbe @pulumi-bot @mjeffryes